Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Simplify command line for scraping datasets #186

Merged
merged 4 commits into from
Jan 17, 2025
Merged

refactor: Simplify command line for scraping datasets #186

merged 4 commits into from
Jan 17, 2025

Conversation

yoomlam
Copy link
Contributor

@yoomlam yoomlam commented Jan 17, 2025

Changes

Use a single entrypoint scrapy-runner script for all scrapy spiders.

Testing

Ensure no 'log_count/ERROR' for each command:

make scrape-ca-public-charge
make scrape-edd-web
make scrape-la-county-policy
make scrape-irs-web

@yoomlam yoomlam marked this pull request as ready for review January 17, 2025 21:31
@github-actions GitHub Actions
Copy link

github-actions bot commented Jan 17, 2025 •
edited
Loading

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
2942 2638 90% 80% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: ba4d89c by action🐍

@yoomlam yoomlam force-pushed the yl/simplify-cmdline-ingestion branch from 3430356 to 38a6d05 Compare January 17, 2025 21:37
fg-nava
fg-nava reviewed Jan 17, 2025
View reviewed changes
app/src/ingestion/scrapy_runner.py
Comment on lines +74 to +76
parser = argparse.ArgumentParser()
parser.add_argument("dataset")
parser.add_argument("--debug", action="store_true")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Help text generation can be useful in the future for usage as complexity of the CLI app increases

@yoomlam yoomlam force-pushed the yl/simplify-cmdline-ingestion branch from a2b095d to c1ae5f5 Compare January 17, 2025 21:58
@yoomlam yoomlam merged commit 151ceb4 into main Jan 17, 2025
4 checks passed
@yoomlam yoomlam deleted the yl/simplify-cmdline-ingestion branch January 17, 2025 23:43
This was referenced Jan 21, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fg-nava fg-nava fg-nava approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yoomlam @fg-nava